Add min_pool_size, Add default value of should_shuffle #70
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| DataProvider* DataProvider::create(const DataConfig& config, | ||
| const ModelConfig& modelConfig, | ||
| bool useGpu) { | ||
| return registrar_.createByType(config.type(), config, modelConfig, useGpu); |
There was a problem hiding this comment.
Add ModelConfig in DataProvider::create to get input layer order
reyoung
force-pushed
the
fix_can_over_batch_size
branch
4 times, most recently
from
d20681d
to
78170c3
Compare
| return dp;\ | ||
| });\ | ||
| }) | ||
|
|
|
Also please update the data provider documentation |
reyoung
force-pushed
the
fix_can_over_batch_size
branch
2 times, most recently
from
aab1c00
to
0210938
Compare
|
@emailweixu Update codes. Add chinese docs. Engligh document will be added asap. |
reyoung
force-pushed
the
fix_can_over_batch_size
branch
from
0210938
to
079cc3f
Compare
|
Need to fix test |
reyoung
force-pushed
the
fix_can_over_batch_size
branch
from
fee49a9
to
5b62bb1
Compare
|
@emailweixu The unittest error before, is because we didn't disable shuffle when unittest and this patch set the Also add english documentation. |
reyoung
force-pushed
the
fix_can_over_batch_size
branch
from
5b62bb1
to
e32e474
Compare
reyoung
commented
Sep 18, 2016
| * cache is a data cache strategy, see `cache`_. | ||
| * Init_hook function is invoked once the data provider is initialized, | ||
| see `init_hook`_. | ||
| .. autofunction:: paddle.trainer.PyDataProvider2.provider |
There was a problem hiding this comment.
Here, we use paddle.trainer.PyDataProvider2.provider's comments as documentation.
reyoung
force-pushed
the
fix_can_over_batch_size
branch
from
e32e474
to
18101a9
Compare
* min_pool_size would be infinite by default.
* add unittest for min_pool_size
* Fix bug in can_over_batch_size
* add unittest for can_over_batch_size
* Add DEFINE_PROVIDER_EX
* Add default value of should_shuffle
* When training, the default value of should_shuffle is True.
* When testing, the default value of should_shuffle is False.
* User a set a provider should_shuffle or not by pass it to `@provider`
* should_shuffle can handle a list of value, not just boolean
* Add input order mapping by using name
* Add unittest
* Add check to check input format.
* Default is close for speed reason.
* User could stop train when check error, or continue train without
this train sample.
* use deque instead of vector in generators pool, make erase
generator faster.
* Add chinese/english documentation
* Make should shuffle = false in unittest
* Add python files to depends.
reyoung
force-pushed
the
fix_can_over_batch_size
branch
from
18101a9
to
981d733
Compare
thisjiang
pushed a commit
to thisjiang/Paddle
that referenced
this pull request
Oct 28, 2021
* refactor lower function * refine LoweredFunc code gen * add const support
gglin001
added a commit
to graphcore/Paddle-fork
that referenced
this pull request
Dec 8, 2021
* add paddleIArray * use final inherit, rm data_
wangxicoding
pushed a commit
to wangxicoding/Paddle
that referenced
this pull request
Dec 9, 2021
* update paddlenlp usage * update paddlelsim * update readme Co-authored-by: ceci3 <592712189@qq.com>
zhoutianzi666
pushed a commit
to zhoutianzi666/Paddle
that referenced
this pull request
May 23, 2022
[DOC] Add C/GO API and R demo
danleifeng
added a commit
to danleifeng/Paddle
that referenced
this pull request
Jul 22, 2022
AnnaTrainingG
pushed a commit
to AnnaTrainingG/Paddle
that referenced
this pull request
Sep 19, 2022
Made several changes: - an -> a - Realse->Release - Traning ->Training - Unify application with noun.
zmxdream
pushed a commit
to zmxdream/Paddle
that referenced
this pull request
Feb 10, 2023
qizhaoaoe
pushed a commit
to qizhaoaoe/Paddle
that referenced
this pull request
Mar 3, 2023
qizhaoaoe
pushed a commit
to qizhaoaoe/Paddle
that referenced
this pull request
Mar 3, 2023
This reverts commit 01a4c47.
lizexu123
pushed a commit
to lizexu123/Paddle
that referenced
this pull request
Feb 23, 2024
hanhaowen-mt
pushed a commit
to hanhaowen-mt/Paddle
that referenced
this pull request
Feb 29, 2024
Fridge003
pushed a commit
to Fridge003/Paddle
that referenced
this pull request
Mar 15, 2024
add group_pattern_util.ShardableAxesProvider
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@providerthis train sample.
generator faster.